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I OBJECTIVE 
The objective of this task was to develop an evaluation procedure 


to assess the relative quality of a set of different remote viewing (RV) 


responses, 


Approved For Release 2003/09/09 : CIA-RDP96-00788R001800090001-4 


Approved For Release 2003/09/09 : CIA-RDP96-00788R001800090001-4 


II INTRODUCTION 


In addressing the remote viewing (RV) evaluation question, we 
consider two separate requirements: 
e Absolute evaluation of a single response for a 
single unknown target. 
e Relative evaluation of a set of responses for 
a series of known targets. 
The first of these is of the most interest in an operational setting. As 
part of the RV enhancement task, we have considered this problem in two 
ways. First, by conducting an operational RV session between two calibra- 
tion RV sessions, a tentative a priori assessment of operational efficacy 
can be determined. The evaluation is made on the basis of performance 
during the calibration sessions, and on the basis of adherence to a pre- 


determined session structure. 


A second technique for an a priori evaluation was explored as 
part of the Fiscal Year 1982 program in an audio-linguistic task. This 
task provided indications that careful linguistic analysis, when coupled 
with technical audio analysis, could yield an assessment in the absence 


of knowledge about the target. 


Various techniques have been used in the cee in an attempt to 
solve the relative evaluation problem. The most common of these was the 
simple rank ordering of all responses, as assessed against all possible 
targets used in an experimental series. In this procedure, a judge is 


presented with n RV transcripts and n target sites. His task is to arrange 


SS 


* 
References are listed at the end of this report. 
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the transcripts in order of the best to least match for each of the n 

targets. A simple numerical counting procedure is then used to estimate 
the likelihood that the judge's transcript/target -matches are by chance 
alone.” This early technique contained little systematic structure for 


determining the final order of matches, 


The first step toward systematizing the rank order judging 
procedure was to preprocess the raw data in the transcript by "concep- 
tualizing" both the verbal and the pictorial responses. Conceptualizing 
a transcript requires an analyst to paraphrase the transcript into a list 
of coherent statements. This concept list is then compared and scored 
concept-by-concept to each of the targets in the experiment. The resulting 
scores are averaged for each response, and all responses are rank-ordered 
on the basis of these scores.° This improved analysis procedure was applied 


to a number of experiments within the Technology Transfer Task 


The problem with the above technique is that there are no guide- 
lines as to how the analyst should paraphrase the transéript; furthermore, 
the method in which the concepts are to be assessed against the targets 
remains undefined. The purpose of the Evaluation Task in FY 1982 was to 


identify a procedure that corrected these deficiencies. 
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III ANALYSIS PROTOCOL 


To quantify the analysis procedure, we have divided the task 
into four separate areas: subject response, target/task definition, 
quality assessment, and numerical analysis, Figure 1 is a sample RV 
Assessment Form that has been designed to emphasize the separation of the 


analysis tasks. Each of the parts of the form are described below. 


The subject's response should be prepared for analysis without 
any Knowledge of either the target site or the overall task, The aim of 
this method of response preparation is to reduce a possibly redundant, 
rambling response to a coherent set of concepts. To meet this requirement 
we have developed a set of initial guidelines to the conceptualization 


procedure, 


A concept is defined as a paraphrase of a single idea that has 
been expressed in the RV verbal or drawing response. That coherent idea 
should not be fragmented into component parts. For example, a response 
might be of the form, "I see a large, textured, gray building." The single 
concept that expresses this idea should be "large, textured, gray building,” 
rather than four separate concepts~-one for each word in the phrase. Each 
concept should be entered under the ''Transcript Concept" column in the RV 


Assessment Form, 


For this initial evaluation technique, a particular concept 
should be used only once in the analysis. (Some weighting factor propor- 
tional to concept frequency could be utilized, but, for the initial attempt, 
only unique concepts are used.) If in the construction of the transcript 
concept list a concept later in the transcript is a duplicate of an earlier 
one, it should be so noted by placing the concept number of the original 


concept in the "D" column. 
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Viewer: 
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Begin Time: 


Date: 


Method of Targeting: 


End Time: 
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SAMPLE RV ASSESSMENT FORM 


FIGURE 1 
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To utilize an analysis procedure that is capable of quantitative 
assessment, it is necessary to define, in advance, what the goals of the 
assessment are. In the RV Assessment Form, columns "Element of Target" 
and "Relevance" are provided to clearly define the goal of the analysis. 
In the ideal situation, an RV target should be completely specified in 
advance. A target typically consists of a number of target elements, 
each of which may have varying relevance with regard to the overall RV 
task. For any given target, an independent list of target elements should 
be prepared. The selection of what constitutes a target element is left 
completely to the discretion of the task coordinator. The target element 
must be selected with little regard to task relevance (target element 
relevance is accounted for later). Because an RV target consists, in 
principle, of an essentially infinite number of possible elements, discre- 


tion needs to be exercised in the selection process. 


For each target element identified for the site, the task 
coordinator must define a relevance rating. This rating allows the 
coordinator to tailor the analysis to the task requirements. Table 1 


shows the scale that is used for the target element relevance rating. 


Table 1 


TASK-DEFINED RELEVANCE SCALE FOR TARGET ELEMENTS 


Rating Relevance Scale 


target element of trivial interest 
element minor interest 


element intermediate interest 


element major interest 


1 
2 
3 
4 
5 


> > > PF PE 


element key interest 
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For each transcript concept on the RV Assessment Form (Figure 1), 
the analyst should attempt to find the element on the list of target 
elements that he/she considers to be the best match. The analyst should 
be quite liberal in the concept/element matching .(i.e., the quality of 
the match should be considered at this point in the analysis). If 
he/she is able to identify a target element that might be considered a 
match to the given concept, a 1 is placed in the "p"” (present) column 
on the assessment form. If no element can be identified, a 0 is placed 
in the "p" column, After making a target element identification, the 
selected target element, and its corresponding overall relevance rating 


| 
should be entered in the appropriate columns on the assessment form, 


Having identified a corresponding target Si snout for each con- 
cept, it is now appropriate to assess the quality of the match. The 
quality assessment is done on the basis of how well the single concept 
in question matches the selected target element. The judgement is to be 
made without regard to any other issues, such as eaporvenee of the concept 
to the transcript, or importance (relevance) of the target element to the 
target. Table 2 shows the quality assessment scale that is used for this 
part of the analysis. The appropriate quality score from Table 2 is 
entered in the "Quality" column on the RV Assessment Form for each concept 


for which a matching target element has been identified. 


An intermediate numerical score is computed for each concept 


from the relevance and quality (Tables 1 and 2) evaluation as follows: 


S’=PyxRxQ ; 


tr ott 


where P is the value in the "p"” column (0 or 1); R is ithe relevance 
evaluation; and 0 is the quality assessment. S’ can ajssume values ranging 
between 0 and 25, Table 3 demonstrates how to determine the final score, 


| 
S, for a given value of 8’ for each concept. The conversion table is 
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Table 2 


QUALITY ASSESSMENT SCALE 


Discrimination Scaie- 


Poor description; only one or two aspects of the material match. 


Fair description; a few aspects of the material match, but a 
large ambiguity exists. 


Reasonable description; many aspects of the material match, but 
there remains some ambiguity. 


Good description; a large number of aspects of the material 
matches, but it is possible to conceive of material that 
would be a better match. 


Excellent description; all or nearly all aspects of the 
material match. 


Table 3 


NUMERICAL SCORE CONVERSION TABLE 


Normalized Score 


Oo wOWmwna KD wm FF Ww HY KY OO 


0 
1 
2 
3 
4 
5 
6 
8 
9 
10 
12 
15 


MN HN Ee 
wm oO a 
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used to eliminate the nonuniformly-distributed gaps in scoring numbers 
that occur if one simply uses the product S*. Thus, the final score for 
each concept ranges from 0 to a maximum of 14. This conversion table is 


“ 


provided as part of the RV Assessment Form. 


If an assessment of an individual concept is required, the final 
score for each concept/target-element match can be related to the quality 
assessment scale by using the conversions shown in the third column of 
Table 3 and on the assessment sheet. It should be noted, however, that the 


integer scores are used to simplify the remaining calculations. 


To determine a final evaluation of the complete transcript 
assessed against a given target, a weighted average of concept scores 
is computed. To assist in the calculation of the weighted average, a 
tally box score is provided at the bottom of the RV Assebsment Form. For 
each of the possible scores, 0 through 14, the number of. concepts that 
attained that particular score are counted. For example, if 3 concepts 
were evaluated with a score of 12, a 3 is entered in the) box below the 
12 score. If the frequency of occurrence of score " is f,, then the final 
weighted average is computed by 


Ay a Oe. Se. KSEE, VE. . 
J J J J J 


0.357 AY 


Ax 


The normalized, weighted average score, Als ig entered in the 
weighted average box on the assessment sheet. The weighted average score 


has been normalized to be within the range 


O<A <5.0 - 


To aid in the interpretation of the result, the quality assessment scale 
(Table 2) can be used to assess quality of the match between the whole RV 


response and the given target site. 


At this point in the evaluation protocol, the: following options 


are available, depending on the task requirement: 


9 


Approved For Release 2003/09/09 : CIA-RDP96-00788R001800090001-4 


Approved For Release 2003/09/09 : CIA-RDP96-00788R001800090001-4 


e An n Xn rank ordering on the basis of the weighted 
averages. 


e A simple selection of the best match. 


e A statistical evaluation on a concept-by-concept 
basis. a 


10 
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IV. CONCLUSIONS 


A protocol has been developed to address the relative evaluation 
portion of the overall RV transcript assessment sete As a demonstra- 
tion of the technique, we provide in Appendix A an analysis of a series 
of four remote viewings that were performed as calibrations 

In this series the remote 
viewing products were of relatively high quality, but honetheless require 
a sensitive technique to differentiate because of the Lintlantey of the 
targets and, hence, of the descriptions. (The series was chosen primarily 
for that reason.) Application of the assessment technique resulted in 
the correct blind matching (highest scoring in matches versus cross 


matches) of three of the four. 


Appendix B is a one page, step-by-step procedure for the 


application of this evaluation technique. 


The material in this document thus constitutes an instruction 

| 
manual or protocol for application of a step-by-step procedure for quan- 
titative assessment of the relative target/transcript correlations of a 


series of transcripts matched into a series of targets. 


ll 
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Appendix A 


EVALUATION OF FOUR COORDINATE REMOTE VIEWINGS 


12 


Approved For Release 2003/09/09 : CIA-RDP96-00788R001800090001-4 


Approved For Release 2003/09/09 : CIA-RDP96-00788R001800090001-4 


On 14 December 1981, four coordinate remote viewings were con- 
ducted as calibrations during remote viewing session tar- 
geted on client-designated sites of interest. The calibrations and the 

RV session were conducted at the client's jorganization and 
were witnessed by client representatives. These four calibrations were 
chosen as a test bed for the evaluation procedure for Eis following 
reasons, (1) they were conducted and (2) the 


targets had many similar features, and would thus provide a sensitive 


test of the protocol. 


Figures A-1 through A-4 are the transcripts that were presented 
to the analyst. They are exactly as they were when collected, except that 
the coordinates have been removed, Figures A-5 through A-8 are the 
National Geographic magazine targets that were used during the calibration 
sessions, Finally, the task coordinator provided Tables A-1 through A-4 
as target element relevance scales for the four targets in Figures A-5 
through A-8, This completes the information that was given to the analyst, 
and thus the analysis was carried out blind as to the matching target/ 


transcript pairs, 


Table A-5 is a compilation of the completed work sheets that 
were used by the analyst in this evaluation. They are shown in groups 
by session number, and alphabetized on the four targets, (The task 
coordinator first randomized the transcript order then assigned the session 
number used above.) For each of the transcripts, the analyst simply 
included all phrases and all drawings as concepts, For example, seven 


concepts were found during Session 2, 


All concepts were then analyzed as described in the text. The 
matching target element, its relevance rating, and the eoupueed score are 
shown for all possible combinations of transcript/target pairs in Table 
A-5. The score distributions and their resulting weighted averages are 


also shown in Table A-5, 
13 
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FIGURE A-4(a) TRANSCRIPT 4 
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Table A-1 


TARGET ELEMENT RELEVANCE RATINGS FOR HANGO 


Target Element 


Town 


Relevance 


Cold 


Peninsula 


Rocky 


Vegetation 


Bay 


Table A-2 


TARGET ELEMENT RELEVANCE RATINGS FOR INVERNESS 


Target Element 


City 


River 


Bridge 


5 
4 
3 
2 


River banks 


Vegetation 1 
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Table A-3 


TARGET ELEMENT RELEVANCE RATINGS 
FOR FERNANDO DE NORONHA 


Target Element 


Island 


Surf 
Hills 
Uninhabited 
Mountain peak 
Temperate climate 
Vegetation 


Ocean 


Table A-4 


TARGET ELEMENT RELEVANCE RATINGS 
FOR PUNKAHARJU 


Target Element 


Connect lakes 5 
Town 4 
Bridges 2 
Cold 4 
Vegetation 1 


Islands 
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#002 


Viewer: 


Hango 


Target: 


12:40 


Begin Time: 


14 December 1981 


Date: 


Coordinates 
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Table A-5 (continued) 


#002 


Target: Fernando de Noronha Viewer: 


12:40 


Begin Time: 


14 December 19381 


Date: 


Coorcinztes 


Mothod of Targeting: 


End Time: 
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Class: 
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Transcript Concept 


Land/water interface 


Up and down 
Picture 3 


Cliff 
Fjords 
Coastal city 


Rocky 


i 
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Length of Session: 
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Table A-5 (continued) 


#002 


Viewer: 


Punkaharju 


Target: 


12:40 


Begin Time: 


14 December 1981 


Date: 


Coordinates 


Method of Targeting: 


End Time: 
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Table A-5 (continued) 


#002 


Target: Inverness Viewer: 


15:23 


Begin Time: 


14 December 


Date: 


Coordinate 


Method of Targeting: 


End Time: 
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Buildings 


Session: 


Transcript Concept 


Straight angles 


Picture 1 
Picture 2 
Buildings 


Length of Session: 
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Table A-5 (continued) 


#002 


Target: Hango Viewer: 


15:23 


Begin Time: 


14 December 1981 


Coordinate Date: 


Method of Targeting: 


End Time: 
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Table A-5 (continued) 


#002 


Viewer: 


Fernando de Noronha 


Target: 


15:23 


Begin Time: 


14 Decemb 


Date: 


Coordinate 


Method of Targeting: 


End Time: 
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Table A-5 (continued) 


#002 


Viewer: 


Punkahar ju 


Target: 


15:23 


Begin Time: 
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Weighted Average 


# of Concepts 


Table A-5 (continued) 


#002 


Viewer: 


Inverness 


Target: 


13:13 


Begin Time: 


14 December 1981 


Date: 


Coordinates 


Method of Targeting: 


End Time: 
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#002 


Target: Fernando de Noronha Viewer: 


13:13 


Begin Time: 


14 December 1981 


Date: 


Coordinates 


Method of Targeting: 


End Time: 
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Table A-5 (continued) 


#002 


Viewer: 


Punkharaja 


Target: 


13:13 


Begin Time: 


14 December 1981 


Date: 


Method of Targeting: Coordinates 


End Time; 
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Table A-5 (continued) 


#002 


Viewer: 


Inverness 


Target: 


12:46 


Begin Time: 


14 December 1981 


Date: 


Coordinates 


Method of Targeting: 


End Time: 
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(continued) 


Table A-5 


#002 


Target: Hango Viewer: 


12:46 


Begin Time: 
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Table A-5 (continued) 


#002 


Viewer: 


Fernando de Noronha 


Target: 


12:46 


Begin Time: 


14 December 1981 


Date: 


Coordinates 


Method of Targeting: 


End Time: 
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Table A-5 (continued) 


#002 


Target; __ Punkaharju Viewer: 


12:46 


Begin Time: 


14 December 1981 


Date: 


Coordinates 


Method of Targeting: 


End Time: 
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Table A-5 
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Two types of overall assessment were chosen to emphasize the 
versatility of the evaluation procedure, (1) a simple rank ordering based 
on weighted average scores, and (2) a concept-by-concept, non-parametric, 
statistical technique. Table A-6 shows the results of the first method, 


the rank ordering. For convenience, the correct matches, are underlined. 


Table A-6 


A RANK ORDERING OF WEIGHTED AVERAGES | 


Fernando 
Session/Target Inverness Hango de Noronha 
ale 


IPunkahar ju 


4.13 eee rs 2 
1.12 | 2.42 
1.21 | 2.21 


1.53 0.90 


* 
Scores computed with non-uniform target relevance factors. 


From Table A-6, we see that there were 3 first-place matches 
and 1 fourth-place match. The probability of obtaining 3 of 4 possible 
first-place matches from chance fluctuations alone are less than 0.051. 
The point spread between the best match (Inverness) and the worst match 
(Punkaharju) are in qualitative agreement with a subjective "first look" 


at the quality of the transcripts as well. 


The second analysis determines the significance of the difference 
between the correct concept/target matches and a control set of matches. 
All concept/target matches that are not the correct matches act as an 
internal control set. To avoid any invalid assumptions as to the correct 
parent distribution, a non-parametric statistical test, the Mann-Whitney 


U-Test, was chosen for the andigetss* 
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It is beyond the scope of this report to review the details 
of the Mann-Witney U-Test; thus, only the results are quoted here. The 
probability that the set of correct concept/target matches is statistically 
indistinguishable from the control concept/target matches is less than 


0.071. 


There are a number of additional statistical procedures that 
could be used to analyze the results of this evaluation technique. The 
two cited above, however, represent a spread in complexity that demonstrates 
the internal consistency of the basic evaluation procedure. With only four 
similar RV sessions, the evaluation technique nearly reached the 0.05 level 
of statistical significance with each of the two statistical procedures, 
a result indicating a successful outcome with regard to the overall 


assessment procedure. 
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Appendix B 


SUMMARY OF EVALUATION TECHNIQUE 


Step Action : 
| 
1 Task coordinator defines the evaluation goal. He/she identifies 
target elements and assigns target element epee factors as 
appropriate. 


2 Analyst conceptualizes responses and prepares an RV assessment 
sheet for each response. 


Repeated concepts are noted in the ''D" column. 


4 Copies of the sheets from Item 2 are made; one for each possible 
target used in the analysis. 


FOR EACH POSSIBLE RESPONSE/TARGET COMBINATION: 

5 Identify a target element for each concept not marked in the 
"ptt column; mark a 1 in the "p' column and write the target 
element and its relevance factor from Step 1 in the appropriate 


columns. (Write 0 and blanks if no element can: ‘be found.) 


6 Using Table 3, assign a quality rating for all present (p = 1) 
concept/element combinations. 
7 Compute the score as follows: 
a. Calculate relevance (R) X quality (Q) : 
b. Convert R X Q to an integer between 0 and 14 using 
the conversion table provided. | 


8 Enter the number of concepts that obtained each| possible score 
in the space provided. | 


9 Calculate the weighted average using: 
| 

a, = 0.357 [2 f, Vf, S,/f £, VE, | 

J J oj J J | 


| 
where: S, is the score and f, is the number of| concepts 
that obtained score 


OC ae a ee 


10 For each response, rank order the weighted averages. 
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